Conventionally, in decoding a moving image comprised of a plurality of encoded frames or fields, for example when a packet loss occurs for data received from a communication path, or when a bit error occurs upon reading decoded data, the data may not be appropriately decoded, and the damaged image is displayed if the display processing is performed as is.
FIG. 12 is an explanatory diagram illustrating pictures of moving image frames and error generation in MPEG-2.
FIG. 12 illustrates a sequence of frames of moving images in MPEG-2. In MPEG-2, three kinds of pictures are defined: Intra-coded Picture (I-picture), Predictive-coded Picture (P-picture), and Bi-predictive picture (B-picture).
The I-picture is encoded while being closed in the frame and does not refer to another frame (picture). The P-picture performs inter-frame prediction in a forward direction from a frame (picture) in the past, and encodes the prediction error. The B-picture performs bidirectional inter-frame prediction using frames (pictures) in the past and frames in the future and encodes the prediction error.
Whether or not an error occurs in pictures of moving image frames is monitored, and if, for example, an error occurs, the picture 903 that is the last normal picture preceding the picture 905 where the error occurs is kept displayed (called a “freeze state”). A method is employed in which the original display is restarted at the frame timing of the picture 907 from where the image may be recovered. Whether a decoding error occurs or not is checked by monitoring macroblocks and seeing where an error occurs in a picture, and if any macroblock where an error occurs exists, updating the display is stopped.
As shown in FIG. 12, if a picture 901 and a picture 907 are I-pictures, decoding may be performed by using the intra-frame information from their own frames. Thus if no error occurs, updating the display at the frame timing of the picture 907 is easy and common.
Generally, a refresh method using an intra-slice (hereunder, called the “intra-slice refresh method”) is employed when achieving error detection and display control in moving image encoding that achieves low-delay.
The intra-slice refresh method does not use B-pictures in order to avoid delays caused by rearranging the order of the pictures. The processing is performed in units of, for example, a macroblock of 16×16 pixels in a picture. The types of macroblocks include an intra-macroblock for encoding within a picture and an inter-macroblock for encoding between pictures.
A “slice” is a group of given macroblocks (for example, a block comprised of vertical 16 pixels×horizontal 16 pixels) arranged horizontally. In MPEG-2, a specific macroblock group (usually from one to a plurality of rows of macroblocks strips in the horizontal direction, or from one to a plurality of columns of macroblock strips in the vertical direction) within a picture is called an “intra-column.” Macroblocks may also be arranged in a rectangular form.
An “intra-slice” is a group of blocks encoded only by image data in a picture. In the intra-slice refresh method, data in a macroblock line in each P-picture is encoded as an intra-macroblock, and the macroblock line is shifted for each cascaded picture so that the intra-slice circulates all the pictures at a substantially constant cycle, and thereby refreshes all the pictures.
The intra-slice refresh method does not require an I-picture that is a main cause for a delay. Thus, a delay caused by a refresh may be shortened with little deterioration of image quality.
However, if motion compensation is applied to slices other than P-picture slices by using a vector without a limitation, a macroblock refreshed by an intra-slice refers to the position of a picture before intra-slice refreshing is applied even if an intra-slice periodically circulates all the pictures. Thus, an error may be propagated spatially and at worst the error screen may continue to be displayed.
In order to reduce or avoid this spatial error propagation, there is the following error detection and display control method.
The error detection and display control method prevents an error from being transmitted by prohibiting motion compensation using a non-zero motion vector for a given period. This prevents an error by motion compensation in one block from moving spatially or increasing.
Unlike MPEG-2/4, the AVC/H.264 standard employs “Exponential-Golomb coding” which does not require a variable length code table.
In many cases, the Exponential-Golomb code may be decoded even when 1 bit of 0 and 1 are reversed, therefore a bit error is difficult to detect. Moreover, even when an error occurs, decoding processing continues for a certain period and an error is detected after a certain amount of time. Thus the position where the bit error is detected and the position where the error is actually generated may often differ. As explained above, regarding detecting an error and controlling a display by a moving image decoding apparatus that conforms to the AVC/H.264 standard, conventionally no apparatus or method exists that achieves low-delay.
FIG. 13 is a figure illustrating an example of error detection and a display control upon decoding a moving image based on the AVC/H.264 standard.
FIG. 13A illustrates a moving image only comprised of P-pictures that include intra-macroblocks. The position of an intra-macroblock (intra-MB) 911 in each of the continuous images sequentially shifts from the upper to the lower parts of the pictures (in the direction of the arrow YA) as time passes. When, the intra-MB 911 reaches the bottom of the picture, the intra-MB911 moves to the top in the next picture.
The positions of intra-MB 911 and intra-MB 913 periodically shift. The cycle in which the position of an intra-MB shifts in a picture is called a “refresh cycle”. For example, in FIG. 13A and FIG. 13B, the refresh cycle is a period of 8 frames.
The encoded data in the next picture located in the same positions as those of intra-MB 911 and intra-MB 913 is not data encoded by intra-frame prediction, but by inter-frame forward direction prediction. Thus, the image may be restored by using an intra-MB immediately before. Therefore, a set of encoded data for one refresh cycle enables decoding of the picture and the subsequent pictures.
At timings of the first intra-MB in each refresh cycle, 921,923, 925, 927, and 931, 933, 935, 937 illustrated in FIGS. 13A and 13B, the moving image decoding apparatus receives parameters of header information on AVC/H.264 encoded data.
The parameter includes information such as a sequence parameter set (SPS), a picture parameter set (PPS), and supplemental enhancement information (SEI).
The SPS is information for defining a parameter for an entire encoded sequence. The PPS is information for defining a parameter of a picture. The SEI is not required for the decoding process itself but is used for supplemental information on decoding processes as information on image freezing/releasing, and recovery points.
In the AVC/H.264 standard, at each refresh cycle, the restore point is indicated by using a recovery point from the SEI that includes the data for the decoding process, and thus the initiation of correct decoding from the restore point can be enabled.
FIG. 13A illustrates an overview when an error occurs in an Active area, while FIG. 13B illustrates an overview when an error occurs in a Negative area.
The Active area is where refresh processing in a refresh cycle has already been completed. The Negative area is where refresh processing has not been applied yet. In FIG. 13, a shaded area in addition to the intra-MB area indicates an Active area, whereas a white area indicates a Negative area.
In FIG. 13A, an error occurrence point 941 indicates frame timing at which a decoding error occurs within an Active area. The picture 951 is a picture at the frame timing of the error occurrence point 941. A picture 953 is a picture at the frame timing immediately before the error occurrence point 941.
A restore point 943 indicates a frame timing where a picture display is released after continuing to display the picture 953 from the error occurrence point 941. Therefore, if no decoding error occurs between the error occurrence point 941 and the restore point 943, the display may be updated at a picture 955 which is the frame timing next to the restore point 943.
In FIG. 13B, an error occurrence point 961 indicates a frame timing where a decoding error occurs within a Negative area. The picture 971 is a picture of the error occurrence point 961, while a picture 973 is a picture of the frame timing immediately before the error occurrence point 961.
A restore point 963 indicates a frame timing where a display of the picture 973 is released after continuing to display the picture 973 located before the error occurrence point 961 starting from the point 961.
In the above conventional method, judging whether an error occurs in the Active area or the Negative area and controlling the display corresponding to the error occurrence point are not performed. Therefore, when an error occurs in a Negative period (error occurrence point 961), the picture 973 instead of the picture 971 is continuously displayed until after passing the period 969 where the next refresh cycle ends, that is the picture 973 is displayed up to the frame timing of restore point 963.
Then the display is updated at the picture 975 based on the recovery point information included in the SEI of parameter information received at the timing 937.
However, the parameter information on a moving image is received at a timing 935 before the timing 937. Therefore, the timing 935 is actually the restore point where the display of picture 971 may be released in the shortest period. The display may be updated at the picture 977 with the timing 935 being a restore point 965, as long as the error occurrence point is within the Negative area and no error occurs thereafter. A period 967, which extends from the error occurrence point 961 to the nearest restore point 965, is short compared to the period 969, which extends from the error occurrence point to the conventional restore point 963.
In the conventional method, whether an error occurs in a Negative area or an Active area is not judged and only an occurrence of a decoding error is judged. Moreover, a display control is applied to an error that occurs in the Negative area at the similar restore timing as an error that occurs in the Active area. Thus, the display may not be restarted in the shortest period when an error occurs in the Negative area, and so a delay may be caused in restoring the normal display.
Moreover, a decoding apparatus performing intra-slice refreshing of an intra-MB in a fixed circulation direction, (e.g., simply treats the area above an intra-MB as refreshed and an area below an intra-MB as unrefreshed) cannot support an encoding apparatus that operates in a reverse direction (e.g., in which the intra-slice refresh circulates from the lower side of an intra-MB). Conversely, a decoding apparatus performing intra-slice refreshing of an intra-MB in circulation from the lower side cannot support an encoding apparatus performing intra-MB circulation from the upper side. Thus, the decoding apparatuses lack versatility in processing.